10/30/2020

OSM Can-BICS

  1. Develop measures using open data sources (OSM) reflecting the amount and quality of bicycle infrastructure, for all communities in Canada, at the neighbourhood level.

  2. Validate the Can-BICS measures using existing national administrative datasets, including Can-ALE (walking & transit environments) and Census journey-to-work data (bicycling behaviour).

OSM Can-BICS Pilot Cities

  • Highlights from OSM Connect US conference
  • Workflow overview
  • Ground data collection
  • OSM Can-BICS classification for pilot cities:
    • Vancouver
    • Montreal
  • Next steps

OSM Connect US

Highlights from OSM Connect US

Oct 30 - Nov 1 2020 (Tuscon AZ / virtual).

  • Lots of talk about recent changes to curb access and slow streets in response to Covid-19.

Lyft

Highlights from OSM Connect US

  • Lyft collected ground data for 30 metros in USA
    • Clustered sampling strategy
    • Remote sensing and ground data
    • 85% + accuracy
    • Region-to-region heterogeneity
    • Paper

Workflow

Ground data

Ground data collection

  • Make an unbiased statement of data quality and fitness for use for the Can-BICS OSM classification.
  • Collect site-specific training and testing data to evaluate and improve queries classifications.

Ground sample

  • 15 cities across Canada
    • Large: Edmonton, Montreal, Ottawa, Vancouver, and Winnipeg (population > 500K)
    • Medium: Halifax, Regina, Saint John (NB), Sherbrooke, and Victoria (pop’n 50 - 500K)
    • Small: Canmore, Cornwall, Courtenay, Whistler, and Whitehorse (pop’n <50 K)

Ground sample

  • Random sample stratified by open data infrastructure type
    • High comfort 2 samples / km (rare and important)
    • Medium and low comfort 1 sample / 10 km
    • Minimum 20 samples per infrastructure type per city
    • No more than 3 samples per km
    • 2222 sample points

Ground data collection

Ground data collection

  • Classification guide developed by Moreno.
  • Four reviewers.
  • All reviewers collected data in all cities.
  • Three rounds of data collection.
  • Met to discuss challenges and work flow after each round.

Ground data quality

  • Points marked for review were double-checked.
  • More than 10% random sample were reviewed by everyone for inter-rater reliability (at least 50 per infrastructure type).

Inter-rater reliability (IRR)

IRR by type.
IRR (%) n
By infrastructure type 84 294
By comfort class 89 294

IRR by infrastructure type

IRR by Can BICS infrastructure type.
Can_BICS_stratum IRR (%) n
Painted Bike Lane 96.2 65
Multi-Use Path 82.8 66
Cycle Track 80.8 46
OSM Only 79.4 67
Local Street Bikeway 79.0 50

IRR by confidence

IRR by confidence (observations marked for review).
Marked for review IRR (%) n
FALSE 92.6 164
TRUE 73.2 130
  • all points marked for review were manually reviewed.
  • in context (i.e., reviewed nearby points on the map).
  • an ongoing process - any ground data mistakes can be reviewed and fixed.

Training / testing split

  • 70 / 30 %.
  • Only training data used so far.
  • Testing data held back for final evaluation.

OSM Classification

Vancouver

Vancouver: Accuracy overall

Table 3. Overall classification accuracy statistics.
Statistic Value
Accuracy 0.72
Kappa 0.65
AccuracyLower 0.65
AccuracyUpper 0.78
AccuracyNull 0.29
AccuracyPValue 0.00
McnemarPValue NaN

Accuracy by class

  • Precision: The probability that a classified feature actually belongs to the class. Also known as user’s accuracy. Calculated as the proportion of classified positives that are true positives.
  • Recall: The proportion of the class that is being detected in the classification. Also known as producer’s accuracy. Calculated as the proportion of classified positives out of the count of all positives in the ground reference data.

Vancouver: Accuracy by class

Classification accuracy statistics by class.
Bike Path Cycle Track Local Street Bikeway Multi-Use Path Painted Bike Lane None
Precision 0.71 0.69 0.96 0.46 0.52 0.82
Recall 0.45 0.73 0.89 0.76 0.89 0.59
Prevalence 0.11 0.16 0.25 0.10 0.09 0.29

Montreal

Montreal: Accuracy overall

Table 3. Overall classification accuracy statistics.
Statistic Value
Accuracy 0.47
Kappa 0.29
AccuracyLower 0.40
AccuracyUpper 0.54
AccuracyNull 0.41
AccuracyPValue 0.06
McnemarPValue NaN

Montreal: Accuracy by class

Classification accuracy statistics by class.
Bike Path Cycle Track Local Street Bikeway Multi-Use Path Painted Bike Lane None
Precision 0.50 0.29 0.16 0.40 0.61 0.58
Recall 0.04 0.57 0.38 0.08 0.65 0.64
Prevalence 0.12 0.12 0.04 0.14 0.16 0.41

Next steps

Next steps for OSM Can-BICS

  • Refine queries and review ground data.
  • All 15 cities.

Next steps for Can-BIKE

  • End of November: all 15 cities classified.
  • End of December: ready (Can-ALE-style metrics and indices for DAs).
  • Goal: National bike infrastructure spatial dataset and associated indices ready by Christmas.
  • January: Validation (compare with Can-ALE and Canada Census Journey to Work).